{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 多智能体基础\n",
    "\n",
    "多智能体条件下，环境是非稳态的，每个智能体都在改变环境，所以转移概率也可能经常变化。\n",
    "\n",
    "* 完全中心化（fully centralized）方法：将多个智能体进行决策当作一个超级智能体在进行决策，即把所有智能体的状态聚合在一起当作一个全局的超级状态，把所有智能体的动作连起来作为一个联合动作，好处是环境仍然是稳态的，收敛性有保证，但是状态空间或者动作空间太大可能导致维度爆炸。\n",
    "* 完全去中心化（fully decentralized）：假设每个智能体都在自身的环境中独立地进行学习，不考虑其他智能体的改变。每个智能体单独采用一个强化算法训练，但是环境非稳态，收敛性不能保证。\n",
    "\n",
    "仍然定义PPO或者其他算法，在训练时，建立多个智能体，每个智能体单独用一个transition表即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 中心化训练去中心化执行(CTDE)\n",
    "\n",
    "是指在训练的时候使用一些单个智能体看不到的全局信息而以达到更好的训练效果，而在执行时不使用这些信息，每个智能体完全根据自己的策略直接动作以达到去中心化执行的效果。中心化训练去中心化执行的算法能够在训练时有效地利用全局信息以达到更好且更稳定的训练效果，同时在进行策略模型推断时可以仅利用局部信息，使得算法具有一定的扩展性。CTDE 可以类比成一个足球队的训练和比赛过程：在训练时，11 个球员可以直接获得教练的指导从而完成球队的整体配合，而教练本身掌握着比赛全局信息，教练的指导也是从整支队、整场比赛的角度进行的；而训练好的 11 个球员在上场比赛时，则根据场上的实时情况直接做出决策，不再有教练的指导。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ❗注意\n",
    "\n",
    "未配置环境，本节代码无法运行，[参考](https://hrl.boyuai.com/chapter/3/%E5%A4%9A%E6%99%BA%E8%83%BD%E4%BD%93%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0%E8%BF%9B%E9%98%B6)，因执行安装环境指令导致conda损坏一次，因此不建议运行。\n",
    "\n",
    "本笔记对代码细节进行了额外标注便于理解思想。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "import rl_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 环境\n",
    "\n",
    "不建议配置，实际上MPE还在维护，但是疑似接口变化比较大，本代码无法配适运行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 仅避免警告，无意义\n",
    "env = []\n",
    "simple_adversary_v3 = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## gumbel-softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def onehot_from_logits(logits, eps=0.01):\n",
    "    ''' 生成最优动作的独热（one-hot）形式 '''\n",
    "    argmax_acs = (logits == logits.max(1, keepdim=True)[0]).float()\n",
    "    # 生成随机动作,转换成独热形式\n",
    "    rand_acs = torch.autograd.Variable(torch.eye(logits.shape[1])[[\n",
    "        np.random.choice(range(logits.shape[1]), size=logits.shape[0])\n",
    "    ]],\n",
    "                                       requires_grad=False).to(logits.device)\n",
    "    # 通过epsilon-贪婪算法来选择用哪个动作\n",
    "    return torch.stack([\n",
    "        argmax_acs[i] if r > eps else rand_acs[i]\n",
    "        for i, r in enumerate(torch.rand(logits.shape[0]))\n",
    "    ])\n",
    "\n",
    "\n",
    "def sample_gumbel(shape, eps=1e-20, tens_type=torch.FloatTensor):\n",
    "    \"\"\"从Gumbel(0,1)分布中采样\"\"\"\n",
    "    U = torch.autograd.Variable(tens_type(*shape).uniform_(),\n",
    "                                requires_grad=False)\n",
    "    return -torch.log(-torch.log(U + eps) + eps)\n",
    "\n",
    "\n",
    "def gumbel_softmax_sample(logits, temperature):\n",
    "    \"\"\" 从Gumbel-Softmax分布中采样，加上从sample_gumbel生成的噪声\"\"\"\n",
    "    y = logits + sample_gumbel(logits.shape, tens_type=type(logits.data)).to(\n",
    "        logits.device)\n",
    "    return F.softmax(y / temperature, dim=1)\n",
    "\n",
    "\n",
    "def gumbel_softmax(logits, temperature=1.0):\n",
    "    \"\"\"从Gumbel-Softmax分布中采样,并进行离散化\"\"\"\n",
    "    y = gumbel_softmax_sample(logits, temperature)\n",
    "    y_hard = onehot_from_logits(y)\n",
    "    y = (y_hard.to(logits.device) - y).detach() + y  # * 式（1）\n",
    "    # * 减一个y再加一个y，仍然是onehot_y，也就是离散的action\n",
    "    # 返回一个y_hard的独热量,但是它的梯度是y,我们既能够得到一个与环境交互的离散动作,又可以\n",
    "    # 正确地反传梯度\n",
    "    # * 我的理解是，对于最后return的y来说，是离散的动作（独热的），但是求梯度时，\n",
    "    # * 在式（1）中，由于前面detach了，没有梯度，因此只对后一个y求梯度，\n",
    "    # * 这个y是来自上一个函数gumbel_softmax_sample得到的y，这样就可以反向传播了\n",
    "    return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DDPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TwoLayerFC(torch.nn.Module):\n",
    "    def __init__(self, num_in, num_out, hidden_dim):\n",
    "        super().__init__()\n",
    "        self.fc1 = torch.nn.Linear(num_in, hidden_dim)\n",
    "        self.fc2 = torch.nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.fc3 = torch.nn.Linear(hidden_dim, num_out)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        return self.fc3(x)\n",
    "\n",
    "\n",
    "class DDPG:\n",
    "    ''' DDPG算法, 参数更新写在MADDPG中 '''\n",
    "    def __init__(self, state_dim, action_dim, critic_input_dim, hidden_dim,\n",
    "                 actor_lr, critic_lr, device):\n",
    "        self.actor = TwoLayerFC(state_dim, action_dim, hidden_dim).to(device)\n",
    "        self.target_actor = TwoLayerFC(state_dim, action_dim, hidden_dim).to(device)\n",
    "        self.critic = TwoLayerFC(critic_input_dim, 1, hidden_dim).to(device)\n",
    "        self.target_critic = TwoLayerFC(critic_input_dim, 1, hidden_dim).to(device)\n",
    "        self.target_critic.load_state_dict(self.critic.state_dict())\n",
    "        self.target_actor.load_state_dict(self.actor.state_dict())\n",
    "        self.actor_optimizer = torch.optim.Adam(self.actor.parameters(), lr=actor_lr)\n",
    "        self.critic_optimizer = torch.optim.Adam(self.critic.parameters(), lr=critic_lr)\n",
    "\n",
    "    def take_action(self, state, explore=False):\n",
    "        action = self.actor(state)\n",
    "        if explore:  # * 探索时，需要训练模型，gumbel_softmax可以反向传播\n",
    "            action = gumbel_softmax(action)\n",
    "        else:  # * 应用时，不需要训练模型，onehot_from_logits直接把连续动作离散，\n",
    "               # * 无法反向传播，但是返回的结果只比gumbel_softmax少一个噪声，差别不大\n",
    "            action = onehot_from_logits(action)\n",
    "        return action.detach().cpu().numpy()[0]\n",
    "\n",
    "    def soft_update(self, net, target_net, tau):\n",
    "        for param_target, param in zip(target_net.parameters(),\n",
    "                                       net.parameters()):\n",
    "            param_target.data.copy_(param_target.data * (1.0 - tau) +\n",
    "                                    param.data * tau)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MADDPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MADDPG:\n",
    "    def __init__(self, env, device, actor_lr, critic_lr, hidden_dim,\n",
    "                 state_dims, action_dims, critic_input_dim, gamma, tau):\n",
    "        self.agents = []\n",
    "        for i in range(len(env.agents)):\n",
    "            self.agents.append(\n",
    "                DDPG(state_dims[i], action_dims[i], critic_input_dim,\n",
    "                     hidden_dim, actor_lr, critic_lr, device))\n",
    "        self.gamma = gamma\n",
    "        self.tau = tau\n",
    "        self.critic_criterion = torch.nn.MSELoss()\n",
    "        self.device = device\n",
    "\n",
    "    @property\n",
    "    def policies(self):\n",
    "        return [agt.actor for agt in self.agents]\n",
    "\n",
    "    @property\n",
    "    def target_policies(self):\n",
    "        return [agt.target_actor for agt in self.agents]\n",
    "\n",
    "    def take_action(self, states, explore):\n",
    "        states = [\n",
    "            torch.tensor([states[i]], dtype=torch.float, device=self.device)\n",
    "            for i in env.agents  # 未写环境可能导致警告\n",
    "        ]\n",
    "        \n",
    "        #  注意：这里的agent.take_action是DDPG算法中的\n",
    "        return [\n",
    "            agent.take_action(state, explore)\n",
    "            for agent, state in zip(self.agents, states)\n",
    "        ]\n",
    "\n",
    "    def update(self, sample, i_agent):\n",
    "        obs, act, rew, next_obs, done = sample\n",
    "        cur_agent = self.agents[i_agent]\n",
    "\n",
    "        cur_agent.critic_optimizer.zero_grad()\n",
    "        # 生成所有目标agent策略的动作\n",
    "        all_target_act = [\n",
    "            onehot_from_logits(pi(_next_obs))\n",
    "            for pi, _next_obs in zip(self.target_policies, next_obs)\n",
    "        ]\n",
    "        \n",
    "        target_critic_input = torch.cat((*next_obs, *all_target_act), dim=1)  # 拼接所有观测和目标agent动作\n",
    "        target_critic_value = rew[i_agent].view(-1, 1) + self.gamma * cur_agent.target_critic(\n",
    "                target_critic_input) * (1 - done[i_agent].view(-1, 1))\n",
    "        critic_input = torch.cat((*obs, *act), dim=1)\n",
    "        critic_value = cur_agent.critic(critic_input)\n",
    "        critic_loss = self.critic_criterion(critic_value, target_critic_value.detach())\n",
    "        critic_loss.backward()\n",
    "        cur_agent.critic_optimizer.step()\n",
    "\n",
    "        cur_agent.actor_optimizer.zero_grad()\n",
    "        cur_actor_out = cur_agent.actor(obs[i_agent])\n",
    "        cur_act_vf_in = gumbel_softmax(cur_actor_out)\n",
    "        all_actor_acs = []\n",
    "        for i, (pi, _obs) in enumerate(zip(self.policies, obs)):\n",
    "            if i == i_agent:  # 如果当前循环对应当前agent，添加当前动作\n",
    "                all_actor_acs.append(cur_act_vf_in)\n",
    "            else:  # 如果当前循环并不对应当前agent而是别的agent，用obs生成他们的动作，保存到一起\n",
    "                all_actor_acs.append(onehot_from_logits(pi(_obs)))\n",
    "        vf_in = torch.cat((*obs, *all_actor_acs), dim=1)\n",
    "        actor_loss = -cur_agent.critic(vf_in).mean()\n",
    "        actor_loss += (cur_actor_out**2).mean() * 1e-3\n",
    "        actor_loss.backward()\n",
    "        cur_agent.actor_optimizer.step()\n",
    "\n",
    "    def update_all_targets(self):\n",
    "        for agt in self.agents:\n",
    "            agt.soft_update(agt.actor, agt.target_actor, self.tau)\n",
    "            agt.soft_update(agt.critic, agt.target_critic, self.tau)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 初始化参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_episodes = 5000\n",
    "episode_length = 25  # 每条序列的最大长度\n",
    "buffer_size = 100000\n",
    "hidden_dim = 64\n",
    "actor_lr = 1e-2\n",
    "critic_lr = 1e-2\n",
    "gamma = 0.95\n",
    "tau = 1e-2\n",
    "batch_size = 1024\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)\n",
    "update_interval = 100\n",
    "minimal_size = 4000\n",
    "\n",
    "observations, infos = env.reset()\n",
    "replay_buffer = rl_utils.ReplayBuffer(buffer_size)\n",
    "\n",
    "state_dims = []\n",
    "action_dims = []\n",
    "\n",
    "action_dims = [i.n for i in env.action_spaces.values()]\n",
    "state_dims = [i.shape[0] for i in env.observation_spaces.values()]\n",
    "\n",
    "critic_input_dim = sum(state_dims) + sum(action_dims)  # 总维度43\n",
    "\n",
    "maddpg = MADDPG(env, device, actor_lr, critic_lr, hidden_dim, state_dims,\n",
    "                action_dims, critic_input_dim, gamma, tau)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 评估和训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(maddpg, n_episode=10, episode_length=25):\n",
    "    # 对学习的策略进行评估,此时不会进行探索\n",
    "    env = simple_adversary_v3.parallel_env(render_mode=\"human\")\n",
    "    returns = np.zeros(len(env.agents))\n",
    "    for _ in range(n_episode):\n",
    "        obs, info = env.reset()\n",
    "        for t_i in range(episode_length):\n",
    "            actions = maddpg.take_action(obs, explore=False)\n",
    "            obs, rew, done, truncated, info = env.step(actions)\n",
    "            rew = np.array(rew)\n",
    "            returns += rew / n_episode\n",
    "    return returns.tolist()\n",
    "\n",
    "\n",
    "return_list = []  # 记录每一轮的回报（return）\n",
    "total_step = 0\n",
    "for i_episode in range(num_episodes):\n",
    "    state, info = env.reset()\n",
    "    # ep_returns = np.zeros(len(env.agents))\n",
    "    for e_i in range(episode_length):\n",
    "        # actions是一个矩阵，每一行表示一个智能体的动作集\n",
    "        actions = maddpg.take_action(state, explore=True)\n",
    "        next_state, reward, done, truncated, _ = env.step(actions)\n",
    "        replay_buffer.add(state, actions, reward, next_state, done, truncated)\n",
    "        state = next_state\n",
    "\n",
    "        total_step += 1\n",
    "        if replay_buffer.size() >= minimal_size and total_step % update_interval == 0:\n",
    "            sample = replay_buffer.sample(batch_size)\n",
    "\n",
    "            # 👇下面这个函数的操作比较难解释，提示：输入网络的维度是状态长度，相当于特征，具体过程看后面\n",
    "            def stack_array(x):\n",
    "                rearranged = [[sub_x[i] for sub_x in x]\n",
    "                              for i in range(len(x[0]))]\n",
    "                return [\n",
    "                    torch.FloatTensor(np.vstack(aa)).to(device)\n",
    "                    for aa in rearranged\n",
    "                ]\n",
    "\n",
    "            sample = [stack_array(x) for x in sample]\n",
    "            for a_i in range(len(env.agents)):\n",
    "                maddpg.update(sample, a_i)\n",
    "            maddpg.update_all_targets()  # 软更新来的\n",
    "    if (i_episode + 1) % 100 == 0:\n",
    "        ep_returns = evaluate(maddpg, n_episode=100)\n",
    "        return_list.append(ep_returns)\n",
    "        print(f\"Episode: {i_episode+1}, {ep_returns}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 关于stack_array函数的解释"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stack_array(x):\n",
    "    rearranged = [[sub_x[i] for sub_x in x]\n",
    "                    for i in range(len(x[0]))]\n",
    "    return [\n",
    "        torch.FloatTensor(np.vstack(aa))\n",
    "        for aa in rearranged\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 环境给出的状态形式，每行代表某一时刻三个智能体的观测，这里假设是随机抽了三个，即行数\n",
    "x = [[np.array([1, 2]), np.array([3, 4]), np.array([3, 5])],\n",
    "     [np.array([4, 2]), np.array([0, 7]), np.array([5, 5])],\n",
    "     [np.array([1, 5]), np.array([4, 7]), np.array([6, 2])],]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[array([1, 2]), array([4, 2]), array([1, 5])],\n",
       " [array([3, 4]), array([0, 7]), array([4, 7])],\n",
       " [array([3, 5]), array([5, 5]), array([6, 2])]]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 第一步\n",
    "[[sub_x[i] for sub_x in x] for i in range(len(x[0]))]\n",
    "\n",
    "# 其实就是转置，但是列表不能转置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa = [[sub_x[i] for sub_x in x] for i in range(len(x[0]))][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([1, 2]), array([4, 2]), array([1, 5])]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [3],\n",
       "       [5]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.vstack(aa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([[1., 2.],\n",
       "         [4., 2.],\n",
       "         [1., 5.]]),\n",
       " tensor([[3., 4.],\n",
       "         [0., 7.],\n",
       "         [4., 7.]]),\n",
       " tensor([[3., 5.],\n",
       "         [5., 5.],\n",
       "         [6., 2.]])]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stack_array(x)  # 把array拆了，倒数第二维变成torch.tensor\n",
    "# 此时第一个输入的样本，即第一个tensor，就是该批次中，第一个智能体所碰到的全部当前状态\n",
    "# 第二个tensor，就是第二个智能体碰到的全部当前状态\n",
    "# 里面tensor部分就和单智能体一样了，参考后面的单智能体输出结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 单智能体的输出过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[-0.9915461 , -0.12975487, -1.7433108 ],\n",
       "        [-0.9959213 ,  0.09022571,  3.2651248 ],\n",
       "        [-0.98676467, -0.1621588 , -1.6151077 ]], dtype=float32),\n",
       " array([[-0.48415332],\n",
       "        [-0.28630628],\n",
       "        [ 0.59497711]]),\n",
       " array([ -9.37310467, -10.37627635,  -9.13395216]),\n",
       " array([[-0.99940634, -0.03445243, -1.91325   ],\n",
       "        [-0.9972526 , -0.07407591,  3.289848  ],\n",
       "        [-0.9967613 , -0.08041708, -1.6474802 ]], dtype=float32),\n",
       " (False, False, False),\n",
       " (False, False, False))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "replay_buffer.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b_s, b_a, b_r, b_ns, b_d, b_t = replay_buffer.sample(3)\n",
    "transition_dict = {'states': b_s, 'actions': b_a, 'next_states': b_ns, \n",
    "                    'rewards': b_r, 'dones': b_d, 'truncated': b_t}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.99729466,  0.07350754,  2.942201  ],\n",
       "       [-0.9995337 , -0.03053527, -2.6963842 ],\n",
       "       [-0.9931456 , -0.1168839 , -8.        ]], dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "b_s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.99729466,  0.07350754,  2.942201  ],\n",
       "       [-0.9995337 , -0.03053527, -2.6963842 ],\n",
       "       [-0.9931456 , -0.1168839 , -8.        ]], dtype=float32)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "transition_dict['states']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.9973,  0.0735,  2.9422],\n",
       "        [-0.9995, -0.0305, -2.6964],\n",
       "        [-0.9931, -0.1169, -8.0000]])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "torch.tensor(transition_dict['states'], dtype=torch.float)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
